Give the chestnut is bothering to another or annoy him, use must come from the use of chestnuts or better yet castanets made of the wood of the chestnut tree which can be a pretty nasty noise.